Constraint Grammar is a hand-crafted Transformer

Syvät neuroverkot (DNN) ja lingvistiset säännöt ovat tällä hetkellä luonnollisen kielen käsittelyteknologioiden ääripäitä. Aina viime aikoihin asti on ollut epäselvää kuinka näitä teknologioita voitaisiin yhdistää. Sen vuoksi näitä teknologioita on tutkittu lähes täysin toisistaan erillään olevissa tutkimusyhteisöissä. Muistutan tässä artikkelissa ensimmäiseksi siitä että sekä sekä Rajoitekieliopilla (CG) että tavallisilla rekurrenteilla neuroverkoilla (RNN) on äärellistilaisia ominaisuuksia. Sitten suhteutan CG:n Google Transformer-arkkitehtuuriin (jossa käytetään kahdenlaista attention-mekanismia) sekä argumentoin, että näiden näennäisesti toisistaan riippumattomien arkkitehtuurien välillä on merkittäviä samankaltaisuuksia.Deep neural networks (DNN) and linguistic rules are currently the opposite ends in the scale for NLP technologies. Until recently, it has not been known how to combine these technologies most effectively. Therefore, the technologies have been the object of almost disjoint research communities. In this presentation, I first recall that both Constraint Grammar (CG) and vanilla RNNs have finite-state properties. Then I relate CG to Google’s Transformer architecture (with two kinds of attention) and argue that there are significant similarities between these two seemingly unrelated architectures.Peer reviewe

Approximating dependency grammars through intersection of regular languages

Springer; 3-540-24318-6;Peer reviewe

Toward a widely usable finite-state morphology workbench for less studied languages, 1: Desiderata

Most of the world’s languages lack electronic word form dictionaries. The linguists who gather such dictionaries could be helped with an efficient morphology workbench that adapts to different environments and uses. A widely usable workbench could be characterized, ideally, as generally applicable, extensible, and freely available (GEA). It seems that such a solution could be implemented in the framework of finite-state methods. The current work defines the GEA desiderata and starts a series of articles concerning these desiderata in finite- state morphology. Subsequent parts will review the state of the art and present an action plan toward creating a widely usable finite-state morphology workbench.Most of the world’s languages lack electronic word form dictionaries. The linguists who gather such dictionaries could be helped with an efficient morphology workbench that adapts to different environments and uses. A widely usable workbench could be characterized, ideally, as generally applicable, extensible, and freely available (GEA). It seems that such a solution could be implemented in the framework of finite-state methods. The current work defines the GEA desiderata and starts a series of articles concerning these desiderata in finite- state morphology. Subsequent parts will review the state of the art and present an action plan toward creating a widely usable finite-state morphology workbench.Peer reviewe

Kaksi tapaa Penn Treebank -puupankin suluttamiseksi

The trees in the Penn Treebank have a standard representation that involves complete balanced bracketing. In this article, an alternative for this standard representation of the tree bank is proposed. The proposed representation for the trees is loss-less, but it reduces the total number of brackets by 28%. This is possible by omitting the redundant pairs of special brackets that encode initial and final embedding, using a technique proposed by Krauwer and des Tombe (1981). In terms of the paired brackets, the maximum nesting depth in sentences decreases by 78%. The 99.9% coverage is achieved with only five non-top levels of paired brackets. The observed shallowness of the reduced bracketing suggests that finite-state based methods for parsing and searching could be a feasible option for tree bank processing.Peer reviewe

On Dependency Analysis via Contractions and Weighted FSTs

Arc contractions in syntactic dependency graphs can be used to decide which graphs are trees. The paper observes that these contractions can be expressed with weighted finite-state transducers (weighted FST) that operate on string-encoded trees. The observation gives rise to a finite-state parsing algorithm that computes the parse forest and extracts the best parses from it. The algorithm is customizable to functional and bilexical dependency parsing, and it can be extended to non-projective parsing via a multi-planar encoding with prior results on high recall. Our experiments support an analysis of projective parsing according to which the worst-case time complexity of the algorithm is quadratic to the sentence length, and linear to the overlapping arcs and the number of functional categories of the arcs. The results suggest several interesting directions towards efficient and highprecision dependency parsing that takes advantage of the flexibility and the demonstrated ambiguity-packing capacity of such a parser.Peer reviewe

Merkityn kaksoisnegaation sovellukset

Nested complementation plays an important role in expressing counter- i.e. star-free and first-order definable languages and their hierarchies. In addition, methods that compile phonological rules into finite-state networks use double-nested complementation or "double negation". This paper reviews how the double-nested complementation extends to a relatively new operation, generalized restriction (GR), coined by the author. ... The paper demonstrates that the GR operation has an interesting potential in expressing regular languages, various kinds of grammars, bimorphisms and relations. This motivates a further study of optimized implementation of the operation.Non peer reviewe

Lingvistiset kieliopit, joilla on on hyvin matala kompleksisuus

We have presented an overview of the FSIG approach and related FSIG gram- mars to issues of very low complexity and parsing strategy. We ended up with serious optimism according to which most FSIG grammars could be decom- posed in a reasonable way and then processed efficiently.Peer reviewe

Bounded-Depth High-Coverage Search Space for Noncrossing Parses

Volume: Proceeding volume: 13A recently proposed encoding for noncrossing digraphs can be used to implement generic inference over families of these digraphs and to carry out first-order factored dependency parsing. It is now shown that the recent proposal can be substantially streamlined without information loss. The improved encoding is less dependent on hierarchical processing and it gives rise to a high-coverage bounded-depth approximation of the space of non- crossing digraphs. This subset is presented elegantly by a finite-state machine that recognizes an infinite set of encoded graphs. The set includes more than 99.99% of the 0.6 million noncrossing graphs obtained from the UDv2 treebanks through planarisation. Rather than taking the low probability of the residual as a flat rate, it can be modelled with a joint probability distribution that is factorised into two underlying stochastic processes – the sentence length distribution and the related conditional distribution for deep nesting. This model points out that deep nesting in the streamlined code requires extreme sentence lengths. High depth is categorically out in common sentence lengths but emerges slowly at infrequent lengths that prompt further inquiry.A recently proposed encoding for non- crossing digraphs can be used to imple- ment generic inference over families of these digraphs and to carry out first-order factored dependency parsing. It is now shown that the recent proposal can be substantially streamlined without information loss. The improved encoding is less dependent on hierarchical processing and it gives rise to a high-coverage bounded-depth approximation of the space of non- crossing digraphs. This subset is presented elegantly by a finite-state machine that recognises an infinite set of encoded graphs. The set includes more than 99.99% of the 0.6 million noncrossing graphs obtained from the UDv2 treebanks through planarisation. Rather than taking the low probability of the residual as a flat rate, it can be modelled with a joint probability distribution that is factorised into two underlying stochastic processes – the sentence length distribution and the related conditional distribution for deep nesting. This model points out that deep nesting in the streamlined code requires extreme sentence lengths. High depth is categorically out in common sentence lengths but emerges slowly at infrequent lengths that prompt further inquiry.Peer reviewe

A hierarchy of mildly context sensitive dependency grammar

The paper presents Colored Multiplanar Link Grammars (CMLG). These grammars are reducible to extended right-linear S-grammars (Wartena 2001) where the storage type S is a concatenation of c pushdowns. The number of colors available in these grammars induces a hierarchy of Classes of CMLGs. By fixing also another parameter in CMLGs, namely the bound t for non-projectivity depth, we get c-Colored t-Non-projective Dependency Grammars (CNDG) that generate acyclic dependency graphs. Thus, CNDGs form a two-dimensional hier- archy of dependency grammars. A part of this hierarchy is mildly context-sensitive and non-projective.The paper presents Colored Multiplanar Link Grammars (CMLG). These grammars are reducible to extended right-linear S-grammars (Wartena 2001) where the storage type S is a concatenation of c pushdowns. The number of colors available in these grammars induces a hierarchy of Classes of CMLGs. By fixing also another parameter in CMLGs, namely the bound t for non-projectivity depth, we get c-Colored t-Non-projective Dependency Grammars (CNDG) that generate acyclic dependency graphs. Thus, CNDGs form a two-dimensional hier- archy of dependency grammars. A part of this hierarchy is mildly context-sensitive and non-projective.Peer reviewe

Monitasoisuus - malli puupankeissa olevia dependenssirakenteita varten

Cited several times. E.g. 1. Marco Kuhlmann & Joakim Nivre: Mildly non-projective dependency structures. In the Proceedings of the COLING/ACL on Main conference poster sessions, p. 507--514. In series COLING-ACL '06. Sydney, Australia, 2006. 2. Carlos Gómez-Rodriguez and Joakim Nivre: A transition-based for 2-Planar Dependency Structures. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1492--1501, Uppsala, Sweden, 11-16 July 2010. ACL 3. Marco Kuhlmann. Dependency Structures and Lexicalized Grammars. An Algebraic Approach. LNAI 6270. FoLLI Publications on Logic, Language and Information. Springer 2010. 4.Eri kielille tehtyjen puupankkien määrä kasvaa tasaista vauhtia. Huomattava osa viimeaikaisista puupankeista käyttää annotaatiokäytäntöä joka perustuu dependenssisyntaksiin. Esitämme tässä paperissa mallin lingvistisesti adekvaattien dependenssirakenteiden luokille. Malli on testattu Danish Dependency Treebankin avulla. jne...The number of treebanks available for different languages is growing steadily. A considerable portion of the recent treebanks use annotation schemes that are based on dependency syntax. In this paper, we give a model for linguistically adequate classes of dependency structures in treebanks. Our model is tested using the Danish Dependency Treebank. Lecerf’s projectivity hypothesis assumes a constraint on linear word- order in dependency analyses. Unfortunately, projectivity does not lend itself to adequate treatment of certain non-local syntactic phenomena which are extensively studied in the literature of constituent-based theories such as TG, GB, GPSG, TAG, and LFG. Among these phenomena are scrambling, topicalizations, WH-movements, cleft sentences, discontinuous NPs, and discontinuous negation. a few relaxed models somewhat similar to projectivity have been pro- posed. These include quasi-projectivity, planarity, pseudo-projectivity, meta-projectivity, and polarized dependency grammars. None of the these models is motivated by formal language theory. The current work presents a new word-order model with a clear connection to formal language theory. The model, multiplanarity with a bounded number of planes, is based on planarity, which is itself a generalization of projectivity.Peer reviewe